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AbStfdCt Objective: Emergency department crowding threatens quality and access to health care, and a 
method of accurately forecasting near-future crowding should enable novel ways to alleviate the problem. The 
authors sought to implement and validate the previously developed ForecastED discrete event simulation for real- 
time forecasting of emergency department crowding. 

Design and Measurements: The authors conducted a prospective observational study during a three-month 
period (5/1/07-8/1/07) in the adult emergency department of a tertiary care medical center. The authors 
connected the forecasting tool to existing information systems to obtain real-time forecasts of operational data, 
updated every 10 minutes. The outcome measures included the emergency department waiting count, waiting 
time, occupancy level, length of stay, boarding count, boarding time, and ambulance diversion; each forecast 2, 4, 
6, and 8 hours into the future. 

Results: The authors obtained crowding forecasts at 13,239 10-minute intervals, out of 13,248 possible (99.9%). The 
R 2 values for predicting operational data 8 hours into the future, with 95% confidence intervals, were 0.27 (0.26, 
0.29) for waiting count, 0.11 (0.10, 0.12) for waiting time, 0.57 (0.55, 0.58) for occupancy level, 0.69 (0.68, 0.70) for 
length of stay, 0.61 (0.59, 0.62) for boarding count, and 0.53 (0.51, 0.54) for boarding time. The area under the 
receiver operating characteristic curve for predicting ambulance diversion 8 hours into the future, with 95% 
confidence intervals, was 0.85 (0.84, 0.86). 

Conclusions: The ForecastED tool provides accurate forecasts of several input, throughput, and output measures 
of crowding up to 8 hours into the future. The real-time deployment of the system should be feasible at other 
emergency departments that have six patient-level variables available through information systems. 
■ J Am Med Inform Assoc. 2009;16:338-345. DOI 10.1197/jamia.M2772. 



Introduction 

The Emergency Medical Treatment and Active Labor Act 
(EMTALA), 1 passed in 1986 by the United States Congress 
as part of the Consolidated Omnibus Budget Reconciliation 
Act, mandates that all patients presenting to an emergency 
department (ED) must be screened and stabilized, regard- 
less of medical condition or ability to pay. Emergency 
medical services are the only form of health care in the 
United States legally guaranteed to be accessible, providing 
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an important safety net. 2-3 A seemingly clairvoyant article 
published in 1958 recommended that the number of EDs 
should increase in preparation for future demand. 4 Unfor- 
tunately, between 1990 and 2005 the annual number of ED 
visits increased from 87 million to 115 million, while the 
number of EDs decreased from 5,172 to 4,611. 5 Due to the 
growing problem of crowding, the Institute of Medicine 
found in 2006 that American EDs are "at the breaking 
point." 6 Prior research has shown that the consequences of 
ED crowding, which include delayed treatment, 7-8 patient 
elopement, 9-10 prolonged transport, 11-12 increased mor- 
tality, 13-14 and financial losses, 15-16 are numerous and affect 
the entire health care system. 

Background 

The ability to forecast near-future ED crowding should 
enable new strategies of coping with the problem. For 
instance, a tool that accurately describes operating condi- 
tions several hours into the future might allow for just-in- 
time dynamic resource mobilization or coordination of 
primary care, hospital, and ED processes. Several measures 
of ED crowding have been proposed, including the Emer- 
gency Department Work Index (EDWIN), 17 the Real-time 
Emergency Analysis of Demand Indicators (READI), 18 
the National Emergency Department Overcrowding Scale 
(NEDOCS), 19 the Emergency Department Crowding Scale 
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(EDCS), 20 and the Work Score. 21 These measures can accu- 
rately reflect present ED crowding, although they do not 
attempt to forecast future ED crowding. 22-23 
Reports in the literature have described efforts to forecast 
ED crowding using techniques such as time series regres- 
sion, 24 deterministic modeling by differential equations, 25 
and discrete event simulation. 26 In the latter, we described 
the development and retrospective validation of ForecastED, 
a simulation-based tool to forecast near-future ED crowd- 
ing. 26 We based the decision to use discrete event simulation 
on the following rationale: Crowding is a complex phenom- 
enon that can be summarized by numerous different mea- 
sures, 27-28 such as the number of waiting patients, boarding 
patients, or occupied beds. Most forecasting techniques 
require the investigator to select a dependent variable before 
model development. By contrast, a discrete event simulation 
can output a detailed list of patients projected to be in the ED 
in the future, and from this information the forecasts of 
many different outcome measures can be derived. 

Research Questions 

To our knowledge, no previous report has demonstrated the 
application of an instrument to forecast ED crowding in real 
time. This research gap must be addressed before a forecast- 
ing system can achieve practical value. We sought to answer 
two questions: First, can the ForecastED tool provide real- 
time crowding forecasts using data from existing ED infor- 
mation systems? Second, how well can it predict near-future 
values of several crowding measures in a live, operational 
setting? 

Methods 

Design, Setting, and Participants 

We prospectively validated the ForecastED tool during a 
three-month period (5/1/07-8/1/07). The study did not 
involve any direct patient contact, and the local Institutional 
Review Board approved the study by expedited review. 
We conducted the study in the adult ED of a tertiary-care, 
urban, academic Level 1 trauma center. The adult ED cares 
for more than 50,000 patients annually. It contains 41 li- 
censed, monitored beds, four of which are designated for 
trauma patients. In addition, four fast-track beds are avail- 
able for patients with minor complaints from 11 am to 11 pm, 
and eight dedicated rooms are available for patients with 
psychiatric complaints. Patients are triaged by the Emer- 
gency Severity Index (ESI), an ordinal score ranging from 
one, for the most urgent patients, to five, for the least urgent 
patients. 29 Institutional policy allows for ambulance diver- 
sion if any of the following criteria are true, and are not 
expected to improve within 1 hour: "(1) all critical care beds 
in the ED are occupied, patients are occupying hallway 
spaces, and at least 10 patients are waiting; (2) an acuity 
level exists that places additional patients at risk; or (3) all 
monitored beds within the ED are full." In practice, the 
administrator on duty initiates ambulance diversion when 
all licensed beds are occupied and at least 10 patients are in 
the waiting room. The ED staff were blinded to the study to 
avoid one potential source of bias. 

We included data from all patients who received care in the 
adult ED during the validation period, with the following 
exceptions: We excluded visits involving purely psychiatric 



complaints because they are treated in a separate unit, and 
little crossover occurs between general-purpose and psychi- 
atric beds. We excluded visits by patients who were dead on 
arrival to the ED, as well as visits by patients who were 
admitted directly to a hospital unit without receiving care in 
the ED, because these patients generally consume little ED 
resources at our institution. Because the study was con- 
ducted in real time, information that identified a patient for 
exclusion was not always available immediately. Such pa- 
tients remained in the study until the time that information 
became available to exclude them. 

System Architecture and Data Collection 

The technical details and assumptions of the core Fore- 
castED tool have been described in previous work. 26 Briefly, 
the tool is a discrete event simulation that consists of several 
random number distributions that describe processes in- 
cluding patient arrivals, evaluation and treatment, and need 
for hospital admission. 30 These distributions drive system 
behaviors including queuing in the waiting room and await- 
ing inpatient beds. Discrete event simulations of ED patient 
flow have been described previously in the literature, gen- 
erally for evaluating proposed operational changes; 31 " 34 the 
ForecastED tool differs substantially because it was de- 
signed for the specific purpose of real-time forecasting. The 
tool may be considered a computerized model of a virtual 
ED — patients have varying degrees of sickness, form queues 
in the waiting room, receive care in licensed beds — with the 
key difference being that time flows much faster in the 
virtual ED than in the actual ED. This property allows us to 
initialize the virtual ED based on the known state of the 
actual ED, rapidly move through several hours of simulated 
time, and obtain crowding measurements at a desired point 
in the future. The raw simulation output consists of a list of 
patient encounters projected to be ongoing in the future, 
together with information describing the course of each 
individual patient through the ED. This output may be 
translated into forecasts of future operational data, using the 
same technique that a list of current patient encounters can 
be summarized into present operational data. For example, 
to calculate the present waiting room count, one may total 
the number of patients in the physical waiting room of the 
ED; similarly, to forecast the future waiting room count, one 
may total the number of patients in the virtual waiting room 
of the simulation output. 

The ED at the study institution is outfitted with a robust 
electronic whiteboard system, which has been previously 
described in the literature. 35-36 The electronic whiteboard 
provides a central point of access to many different pieces of 
data that describe the minute by-minute operating status of 
the ED. Its roles include tracking patients, monitoring safety, 
and alerting personnel to important issues. Numerous as- 
pects of ED patient data are made available in real time by 
Oracle databases (version lOg, http:/ / www.oracle.com) un- 
derlying the electronic whiteboard system. 
The core ForecastED tool was written in American National 
Standards Institute (ANSI) compliant C with no dependen- 
cies on external libraries; moreover, it is agnostic to the local 
ED information technology environment. To enable real- 
time forecasting of crowding, we developed a wrapper 
program using the Python prograrnming language (version 
2.3.5, http:/ /www.python.org) that interfaces with the existing 
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ForecastED: Crowding Status 



07/23/0711:33:26 



Diversion probability: 



84% 88% 91% 92% 




Figure 1 . Graphical web in- 
terface of the ForecastED output. 
As indicated in the upper right, 
this screenshot was captured on 
a Monday (July 23, 2007) shortly 
after 11:30 am, which is com- 
monly a time of rapid patient 
inflow. Across the top, the fore- 
casts of ambulance diversion 
probability at 1 hour 2 hours, 
and up to 8 hours into the future 
are displayed. Most of the screen- 
shot is occupied by time series 
plots for six crowding measures, 
where the solid line indicates the 
actual values observed during 
the previous 16 hours, and the 
diamonds indicate the forecast 
values for the next 8 hours. 



databases and automates the process of obtaining forecasts. 
The functions of this program were to obtain deidentified 
patient data from local ED information systems, to provide 
these data to ForecastED as input, to record the subsequent 
forecasts, and to display the output using the graphical web 
interface shown in Figure 1. The ED personnel did not have 
access to the interface, which was password-protected; this was 
intended to prevent the forecasting tool from influencing 
decision-making about ambulance diversion, as this would 
confound the results of the validation. Six patient-level vari- 
ables were obtained from ED information systems when need- 
ed 26 : (1) time of initial registration at the ED, (2) time placed 
into an ED treatment bed, (3) time of hospital bed request if 
applicable, (4) time of discharge from the ED facility, (5) triage 
category assigned to the patient, and (6) whether the patient 
left without being seen. 

The wrapper program performed two tasks at regular inter- 
vals: (1) Every day at midnight, the program used the most 
recent four weeks of historical patient data to estimate the 
parameters of each random number distribution needed for 
the simulation. This was intended to keep the simulated 
processes up to-date despite long-term fluctuations that 
might occur in patient demand or local workflow. (2) At 
every 10-minute interval, the program identified the set of 
patients presently in the ED and initialized the ForecastED 
tool accordingly. It used the mean of 1,000 simulation 
replications to obtain forecasts of several crowding mea- 
sures at each hourly interval up to 8 hours into the future. 37 



Outcome Measures 

We obtained forecasts of seven distinct crowding measures 
at 10-minute intervals: (1) waiting count, defined as the 
number of patients in the waiting room; (2) waiting time, 
defined as the average time since presentation among pa- 
tients in the waiting room; (3) occupancy level, defined as 
the total number of patients in ED beds divided by the 
number of licensed treatment beds (this value may exceed 
100% when patients are treated in nonlicensed areas such as 
hallway beds or chairs); (4) length of stay, defined as the 
average time since presentation among all patients in ED 
beds; (5) boarding count, defined as the number of patients 
awaiting hospital admission; (6) boarding time, defined as 
the average time since hospital bed request among patients 
awaiting hospital admission; and (7) probability of ambu- 
lance diversion, defined as a close approximation of the 
local diversion policy using the probability of having 10 
or more patients in the waiting room and an occupancy 
level of at least 100%. These outcome measures were 
identical to those used in the preliminary, retrospective 
validation of ForecastED. 26 

We used the actual outcome measure, from the correspond- 
ing point in the future, as the reference standard for validat- 
ing each forecast. At the time when each forecast was 
recorded, the value of the reference standard was not yet 
known. After the validation period ended, we obtained 
actual values of the six continuous outcome measures from 
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Table 1 ■ Reliability of the Simulation in Forecasting Operational Data* 





2 h Ahead 


4 h Ahead 


6 h Ahead 


8 h Ahead 


Waiting count (R 2 ) 


0.53 (0.52, 0.55) 


0.40 (0.39, 0.42) 


0.32 (0.31, 0.34) 


0.27 (0.26, 0.29) 


Waiting time (R 2 ) 


0.32 (0.29, 0.35) 


0.22 (0.20, 0.24) 


0.15 (0.13,0.17) 


0.11 (0.10, 0.12) 


Occupancy level (R 2 ) 


0.76 (0.75, 0.76) 


0.67 (0.66, 0.68) 


0.61 (0.60, 0.62) 


0.57 (0.55, 0.58) 


Length of stay (R 2 ) 


0.87 (0.87, 0.88) 


0.80 (0.80, 0.81) 


0.74 (0.73, 0.75) 


0.69 (0.68, 0.70) 


Boarding count (R 2 ) 


0.84 (0.84, 0.85) 


0.74 (0.73, 0.75) 


0.67 (0.66, 0.68) 


0.61 (0.59, 0.62) 


Boarding time (R 2 ) 


0.70 (0.69, 0.71) 


0.61 (0.60, 0.62) 


0.56 (0.55, 0.57) 


0.53 (0.51, 0.54) 



The coefficient of determination is presented with lower and upper bounds of the 95% confidence interval in parenthes 



information systems. The local aeromedical service, which 
maintains official records of diversion status independently 
of the ED, provided ambulance diversion log files. 
Statistical Analysis 

We validated the simulation forecasts of each outcome 
measure 2, 4, 6, and 8 hours into the future. We used the 
coefficient of determination (R 2 ) to measure the reliability of 
the simulation forecasts for each continuous outcome mea- 
sure with respect to the reference standard. This statistic 
describes the percentage of variation in the future outcome 
measures explained by the simulation forecasts. We calcu- 
lated the R 2 with 95% confidence intervals (CI) using 250 
iterations of the ordinary bootstrap method. 38 
The values of R 2 would not be affected by the calibration of 
the simulation forecasts; 39 furthermore, because ForecastED 
was not fitted to predict any specific dependent variable in 
the least-square sense, it would have been erroneous to 
assume the residuals were centered around zero. We inves- 
tigated the possible bias by calculating the mean and stan- 
dard deviation of the residual forecasting error for each 
continuous outcome measure. A residual mean differing 
from zero, with respect to the standard deviation, would 
reveal the presence of a systematic bias in the forecasts. 
We calculated the area under the receiver operating charac- 
teristic curve (AUC) to assess the discriminatory power in 
forecasting ambulance diversion status. This statistic sum- 
marizes overall discriminatory power for a binary outcome, 
where a value of 1.0 denotes perfect discrimination and a 
value of 0.5 denotes no discrimination. 40 We calculated the 
AUC with 95% CI using 250 iterations of the ordinary 
bootstrap method. 38 All statistical analyses were conducted 
using R (version 2.3.1, http:/ /www.r-project.org). 

Results 

During the study period, a total of 13,239 10-minute inter- 
vals were observed out of a possible 13,248 (99.9%). Brief 
network downtimes accounted for the missed observations. 
A total of 14,448 visits by 11,539 unique patients occurred in 
the adult ED during the study period, of which 1,348 visits 



were excluded (9.3% total, 1.0% psychiatric, 0.0% dead on 
arrival, 8.3% immediately admitted to the hospital). Females 
represented 54.9% of the total patients, and the median age 
was 39 years. A total of 73.8% of the patients arrived by car, 
17.1% by ambulance, 2.2% by helicopter, and 6.9% by other 
or unknown means. Hospital admissions resulted from 
22.7% of the ED visits. A total of 77 ambulance diversion 
episodes, each lasting an average of 5.4 hours, occurred 
during the study period (18.8% of the total time). 
The reliability of the simulation forecast for each continuous 
outcome measure is presented in Table 1. The forecasts 
explained more than 50% of the variation in the occupancy 
level, length of stay, boarding count, and boarding time up 
to 8 hours into the future. The percentage of future variation 
explained decreased as the length of the forecasting window 
increased. For example, the simulation forecasts of the 
occupancy level had R 2 values of 0.76, 0.67, 0.61, and 0.57, 
respectively, in predicting the actual occupancy level 2, 4, 6, 
and 8 hours into the future. 

The calibration of the simulation forecast for each continu- 
ous outcome measure is presented in Table 2. The residual 
mean had small magnitude, relative to the standard devia- 
tion, for every outcome measure except the boarding time, 
suggesting that the forecasts were unbiased for most mea- 
sures of crowding studied. The model consistently underes- 
timated the boarding time 2, 4, 6, and 8 hours into the future, 
respectively, by -6.6 ± 2.7, -7.3 ± 3.1, -7.7 ± 3.3, and 
—7.8 ± 3.5, demonstrating a systematic bias for this outcome 
measure. 

The receiver operating characteristic curves for discriminat- 
ing future ambulance diversion status are presented in 
Figure 2. The AUC at 2, 4, 6, and 8 hours into the future, 
respectively, was 0.93 (95% CI: 0.93, 0.94), 0.90 (95% CI: 0.90, 
0.91), 0.88 (95% CI: 0.87, 0.88), and 0.85 (95% CI: 0.84, 0.86), 
suggesting good discrimination. To illustrate the response of 
the ForecastED tool to periods of ED crowding, Figure 3 
shows a time series plot of the 6-hour forecast probability of 
ambulance diversion, superimposed on episodes of ambu- 
lance diversion. 



Table 2 ■ Calibration of the Simulation in Forecasting Operational Data* 




2 h Ahead 


4 h Ahead 


6 h Ahead 


8 h Ahead 


Waiting count (# of patients) 


0.0 ± 4.5 


0.9 ± 5.8 


1.6 + 6.5 


2.2 + 7.0 


Waiting time (hours) 


-0.1 + 0.6 


0.1 + 0.9 


0.3 + 1.1 


0.5 + 1.3 


Occupancy level (% of beds) 


2.4 ± 9.6 


2.5 + 11.2 


2.9 + 12.1 


3.3 + 12.9 


Length of stay (hours) 


-0.7+1.0 




-0.8 + 1.5 


-0.8 + 1.6 


Boarding count (# of patients) 


0.3 + 2.5 


0.2 ± 3.1 


0.1 + 3.6 


0.1 ± 3.9 


Boarding time (hours) 


-6.6 ± 2.7 


-7.3 ± 3.1 


-7.7 ± 3.3 


-7.8 + 3.5 



The forecasting residuals are summarized with the mean ± standard deviation. 
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Forecasting length: AUC (95% CI) 

2 hours: 0.93 (0.93, 0.94) 

4 hours: 0.90 (0.90, 0.91) 

6 hours: 0.88 (0.87, 0.88) 
8 hours: 0.85 (0.84, 0.86) 



0.0 0.2 0.4 0.6 0.8 1.0 

1 - specificity 

Figure 2. Receiver operating characteristic curves of the sim- 
ulation forecast of ambulance diversion at 2, 4, 6, and 8 hours into 
the future. The AUC with 95% CI is shown in parentheses. 

Discussion 

We have successfully deployed the ForecastED tool in an 
adult ED. Three technical barriers existed with respect to 
the real-time implementation. First, the tool required the 
timely electronic availability of ED operational data. The 



study was conducted in a setting where an electronic 
whiteboard system was already accepted by faculty and 
staff, 35-36 so this issue was considered resolved at the 
inception of the research. Second, a reliable, automated 
technique was needed to estimate parameters and obtain 
forecasts at scheduled intervals. We developed a Python 
wrapper program to address this need, and the results 
showed appropriate uptime statistics. Third, some mecha- 
nism was needed to instantly disseminate the forecasts to 
interested parties. To accomplish this, we created a graphical 
web interface as shown in Figure 1, which is updated with 
every observation. 

The validation results demonstrate that the ForecastED tool 
can provide real-time, accurate predictions for a variety of 
ED crowding measures. Using information from past and 
present patients, the tool accurately predicted five out of 
seven outcome measures tested, up to 8 hours into the 
future. It fared less well in forecasting aspects related to the 
waiting room, which may suggest the crowding status in 
the waiting room is more volatile than the crowding status 
in other parts of the ED workflow. The forecasts of the 
waiting count and waiting time may be most useful when 
considered 4 hours or less into the future. 
The results suggested that the forecasts for each continuous 
outcome measure were well calibrated for all outcome 
measures except for the boarding time, which showed 
evidence of a systematic bias. This result was consistent with 
our previous observations, and a likely mechanism exists 
within the simulation assumptions that govern allocation of 
inpatient hospital beds. 26 This issue may be resolved by 






Figure 3. Time series plot of the 6-hour forecast of ambulance diversion status. The horizontal axis marks the date and time 
at which the forecast was obtained, and the vertical axis denotes the probability of fulfilling the criteria for ambulance diversion 
6 hours into the future from that point. The signal shown here has been smoothed using cubic splines. Episodes of ambulance 
e marked by the shaded areas. 
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linearly calibrating the forecasts of boarding time according 
to the known bias in the residuals. 

The results also indicated that the forecasts accurately pre- 
dicted the probability of future ambulance diversion status 
up to 8 hours into the future. The time series plot shown in 
Figure 3 illustrates that the predictions closely fit the pattern 
of ambulance diversion in the study setting, with a peak in 
the signal associated with most diversion episodes. A sharp 
rise in the signal tends to precede the start of each diversion 
episode, while a sharp fall in the signal tends to precede the 
end of each diversion episode. 

The screenshot in Figure 1 illustrates a unique capability of 
ForecastED: because a discrete event simulation model fits 
the pattern of individual patient flow, rather than a specific 
dependent variable, it can describe crowding from multiple 
input, throughput, and output perspectives. 27 Thus, it not 
only warns that an ED will be crowded; it may also pinpoint 
why the ED will become crowded, which may be different at 
different times. For example, on a given day crowding may 
be attributed to a rapid inflow of patients that overload the 
waiting room, as shown in Figure 1. On another day 
crowding may be caused by large numbers of boarding 
patients, who remain in the ED for long periods while 
awaiting hospital admission. 

In addition to the graphical web interface described above, 
other methods of distributing forecasting results are possi- 
ble. The tool could interface with a pager system to alert 
on-call providers of severe crowding, for example, or it 
could interface with e-mail systems to distribute alerts. The 
proper method of disseminating warnings of crowding may 
vary between institutions depending on the local culture 
and preferences. Any operational change must be tuned to the 
organizational structure of an institution. Furthermore, some 
precedent exists for neighboring EDs to share operational data 
through regional networks, 41-42 and this precedent could be 
applied to achieve local sharing of forecasting results. This 
would potentially enhance cooperation in areas with multiple 
busy tertiary care centers, particularly level 1 trauma centers. 
One consideration during the prospective implementation 
and deployment of the ForecastED tool was how to clean 
patient data in real time. It is standard practice to identify 
and remove outlier data, when appropriate justification is 
provided 43 We justified the previously stated patient exclu- 
sion criteria on the grounds that not all patients recorded in 
the ED information system participate in the normal ED 
patient flow. The criteria would be straightforward to apply 
retrospectively, but not prospectively. For example, a patient 
with a purely psychiatric complaint would be identified and 
excluded at the beginning of a retrospective data analysis. 
However, a patient who is in the waiting room during the 
prospective study might not be identified, and hence ex- 
cluded, until after the patient is placed into a dedicated 
psychiatric bed. Thus, the patient would affect the prospec- 
tive validation until adequate information existed to mark 
him or her for exclusion. Although we were unable to 
eliminate this challenge, the results suggest this does not 
compromise the utility of the forecasting tool. 
No changes should need to be made to the core ForecastED 
tool when transporting it to other institutions, because it was 
written in the standard C programming language with no 



dependencies on external software. The chief alteration 
required to deploy the tool at other institutions would 
involve changing the Python wrapper program that con- 
nects to ED information systems, since different institutions 
have different database storage schemes. Because the soft- 
ware automatically recalibrates the model parameters every 
day at midnight, no additional changes should be needed at 
other sites. Based on local preferences, the tool could also be 
adapted to forecast outcome measures aside from those used 
in our study. For example, other institutions are likely to have 
ambulance diversion criteria that differ from our study setting. 
No changes would need to be made to the simulation itself, 
and minor changes would need to be made to the code that 
processes the simulation output, to forecast ambulance diver- 
sion according to different criteria. The process of real-time 
system deployment should be repeatable in any ED that has six 
required patient-level variables available electronically. 
Our study was limited in part because it took place over a 
four-month period in the adult ED of a single academic 
institution. Within our site seasonal patterns may exist that 
were not represented in the study period, which could lead 
the forecasting accuracy to deteriorate. Also, other hospitals 
may have differences in organizational structure or patient 
populations, leading to variable performance among sites. 
The present study does not allow for comment on these 
issues, so further validation at our institution and external 
sites will be needed to establish how these findings gener- 
alize with respect to time and place. 

Our study was also limited in part because we made no 
intervention based on the tool. This allowed us to validate 
the forecasting accuracy in a live, operational setting; how- 
ever, the goal of ForecastED is not merely to provide 
information, but to spur action based on this information to 
alleviate ED crowding. The question of how interventions 
triggered by the forecasts would directly impact patient care 
remains a valuable topic for further research. Few reports 
have discussed the prospect of allocating personnel and beds 
on demand, 44 perhaps because the technology to determine 
when to mobilize such resources has generally been unavail- 
able. We conclude the discussion below with two scenarios 
whereby the early warning provided by the ForecastED tool 
could help alleviate the negative effects of ED crowding. 
Our first conjectural scenario considers the possibility of 
flexible staffing, which might be facilitated by operational 
forecasts. Nurses are often understaffed and overworked, 
with ED nurses caring for more than 5 patients simulta- 
neously in some settings 45 Administrators plan staffing 
schedules based on long-term daily and weekly trends in 
patient volume, with the intent of maintaining a patient-to- 
nurse ratio near 4:1. Short-term fluctuations in ED opera- 
tions render this difficult, however. This problem might be 
addressed by scheduling more nurses to staff the ED than 
are needed during average operating conditions; subse- 
quently, when the ForecastED tool anticipates periods of 
lessened need, nurses could be given the option of ending a 
shift early. This strategy avoids the potential morale impact 
of mobilizing off-duty staff to achieve flexible staffing, while 
mitigating the additional costs of keeping the ED overstaffed 
at all times. Using this mechanism, the staffing level could be 
tailored dynamically as needed. 
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Our second conjectural scenario targets the widespread 
problem of ED boarding, which is the practice of retaining 
patients admitted to the hospital in ED beds for extended 
periods until an inpatient bed becomes available. 46 This 
temporarily reduces the effective capacity of an ED, 
because the space where new patients could be treated 
remains occupied. When anticipating the daily demand 
for inpatient beds, hospital administrators must consider 
factors like transfers from other hospitals and operating 
room schedules. They must also consider hospital admis- 
sions originating via the ED, and forecasts generated by 
our tool could assist the accuracy of this process. In this 
way, the ForecastED tool might improve daily discharge 
planning, facilitating coordination between hospital ser- 



Conclusions 

In summary, we have deployed and prospectively vali- 
dated the ForecastED tool, which provides potentially 
useful forecasts of various ED crowding measures up to 8 
hours into the future. In keeping with the principle "you 
can't manage what you can't measure", this may enable 
new, proactive strategies for coping with the ED crowding 
problem. This work may provide a means of protecting 
and strengthening the fragile safety net of the health care 
system. 
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